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FOREWORD 

In  1967,  a  booklet  entitled  Summary  Description  of  Grade  Nine 
Literature  Objectives,  Teat  Items  and  Blueprint  was  issued  to 
Alberta  teachers.  Therein,  cognitive  and  affective  objectives 
in  the  field  of  literature  were  classified  under  the  categories 
of  Knowledge,  Comprehension,  Higher  Mental  Processes,  and  the 
Affective  Domain.  Using  sample  test  items  drawn  from  the  1967 
Literature  Departmental  Examination,  the  present  supplement 
illustrates  the  categories  set  forth  in  the  above  mentioned 
booklet  and,  in  addition,  demonstrates  the  value  of  item  analy- 
sis to  the  teacher  who  seeks  to  develop  his  skill  in  construct- 
ing, evaluating,  and  refining  items  for  valid  and  reliable  tests. 

Each  selected  item,  printed  as  it  appeared  in  the  examination 
booklet,  is  followed  by  item  analysis  data.  The  item  is  then 
evaluated  first  in  terms  of  the  thought  processes  involved  in 
arriving  at  the  correct  answer  and  then  in  terms  of  the  item 
analysis  data.  Because  the  data  are  presented  exactly  as  they 
appear  on  a  computer  print-out,  charts  on  pages  2  through  7  of 
this  pamphlet  demonstrate  how  to  identify,  interpret,  and  use 
the  statistics  included  in  item  analysis. 

The  present  document  represents  only  one  type  of  test  from  among 
the  many  approaches  to  evaluation  effectively  used  by  teachers  in 
the  classroom.  Within  the  area  of  objective  testing,  multiple- 
choice  questioning  is  receiving  increased  emphasis.  The  teacher 
can  develop  this  aspect  of  his  testing  program  by  using  this 
study  to  increase  the  accuracy,  discriminatory  power  and  validity 
of  each  test  item  that  he  uses. 


ii 


PART  I 
ELEMENTS  OF  ITEM  ANALYSIS 

The  item  analysis  data  accompanying  each  item  in  PART  II  of  this 
pamphlet  could  prove  difficult  for  the  teacher  who  is  unfamiliar 
with  the  format  in  which  the  data  is  presented.  Therefore,  the 
following  charts  have  been  constructed  to  prepare  the  teacher  for 
reading  end  using  this  data.  Each  chart  introduces  information 
that  is  complementary  to  that  found  in  the  preceding  chart. 
Chert  I  labels  pnd  defines  the  parts  of  a  multiple-choice  item. 
Chart  2  »nd  Chart  3  identify  and  define  the  elements  of  an  item 
analysis.   Chart  4  sets  forth  limits  of  acceptability  for  each 
element.   Chart  5  illustrates  an  interpretation  related  to  an 
acceptable  item,  while  Chart  6  provides  an  interpretation  related 
to  an  unacceptable  item.  It  is  recommended  that  the  teacher 
carefully  study  the  information  and  interpretations  presented  in 
these  charts  before  proceeding  to  the  detailed  evaluations  found 
in  PART  II. 
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PART  II 
EVALUATIONS 

The  following  evaluations  have  been  compiled  to  reveal  both 
strengths  and  weaknesses  in  sample  multiple-choice  items.  Along 
with  some  of  the  successful  items,  unacceptable  items  were  also 
selected  from  the  1967  Literature  Departmental  Examination  in 
order  to  illustrate  as  many  uses  of  the  item  analysis  as  possible. 
Effort  has  also  been  made  to  choose  items  representative  of  the 
categories  set  forth  in  the  Summary  Description  of  Grade  Nine 
Literature  Objectives,  Test  Items,  and  Blueprint,  1967*  Complete 
statistics  for  each  item  have  been  included  to  facilitate  a 
clearer  comprehension  of  each  evaluation.  The  general  statistics  ■ 
including  the  level  of  difficulty,  Biserial  Correlation,  and  the 
Item  Reliability  Index  -  are  considered  first.  Then  the  detailed 
statistics  are  discussed.   Where  needed,  suggestions  for  revisions 
are  offered. 


EVALUATION  1  —  TEST  ITEM  35 

35.  Each  of  the  following  is  a  sub-class  of  lyric  poetry  EXCEPT 

A.  ode 

B.  elegy 

C .  narrative 

D.  sonnet 


ITEM 

N   OMIT 

DIF 

NR 

NF 

K 

1 

35 

1026    7 

0.469 

0 

3 

118 

188     UPPER 

5 

1 

7 

210     UPPER 

4 

0 

19 

214     UPPER 

3 

2 

31 

196     UPPER 

2 

2 

24 

218     UPPER 

1 

2 

37 

TEST  SCORE  MEANS 

43.8 

Z-SCORE  MEANS 

-0.08 

-0.36 

BISERIAL  CORREL  §   0.375 

ITEM  REL.  INDEX  #  0.187 

2 

CD 

4 

221 

481 

199 

12 

133 

35 

45 

111 

35 

42 

99 

40 

51 

58 

50 

71 

58 

50 

44.1 

51.6 

47.0  ! 

-0.36 

0.33 

-0.10 

This  item  wes  placed  in  the  Knowledge  category  because  the  item  requires 
the  student  to  remember  specific  information  about  the  classes  of  lyric 
poetry. 

The  level  of  difficulty  indicates  that  approximately  47^  of  the  students 
responded  correctly  to  this  item.  This  means  that  the  question  is  of 
pverape  difficulty.  The  Biserial  Correlation  of  0.375  reveals  that  the 
item  discriminated  well  between  those  who  scored  higher  on  the  total 
test  and  those  who  did  not.  The  Item  Reliability  Index  of  0.187  indicates 
that  one  can  depend  on  this  Biserial  Correlation  with  a  fairly  high  degree 
of  confidence. 

All  distractors  (incorrect  alternatives)  functioned  very  well.  Number  2 
wrs  the  strongest  because  it  attracted  122  lower  students  and  only  57  of 
the  upper  students.  The  weakest  distractor  is  alternative  4  which  was 
too  attractive  to  the  upper  students.  Revision  of  this  item  would  prob- 
ably require  attention  to  this  weak  distractor. 


In  the  statistics  for  each  item  to  follow  the  Keyed  Answer  will  be 
circled. 
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EVALUATION  2  —  TEST  ITEK  13 

13.  The  selection  which  emphasizes  the  importance  of  the  Bible  in 
the  upbringing  of  children  is 

A.  God  is  at  the  Anvil 

B.  }ty  Mother's  Voice 

C.  The  Cotter's  Saturday  Night 

D.  The  Happy  Journey  to  Trenton  and  Camden 


ITEM     N   OMIT      DIF 

NR 

NF 

K 

1 

2 

G) 

4 

13   1026   171    0.438 

0 

3 

115 

110 

449 

181 

188     UPPER 

5 

43 

2 

4 

115 

24 

210     UPPER 

4 

31 

14 

11 

119 

35 

214     UPPER 

3 

40 

24 

14 

97 

39 

196     UPPER 

2 

29 

27 

42 

60 

38 

218     UPPER 

1 

28 

48 

39 

58 

45 

TEST  SCORE  MEANS 

40.9 

41.5 

51.4 

46.7 

Z-SCORE  MEANS     -0.16 

-0.65 

-0.54 

0.24 

-0.10 

BISERIAL  CORREL  #  0.335 

ITEM  REL.  INDEX  #  0.166 

This  item  was  designed  to  test  knowledge  of  the  philosophy  underlying 
a  specific  work.   Since  the  student  was  required  only  to  remember  the 
theme  of  the  correct  alternative  in  order  to  answer  the  question 
correctly,  this  item  was  assigned  to  the  Knowledge  category. 

The  difficulty  level,  of  approximately  44^  indicates  that  the  item  was 
somewhat  difficult.  The  Biserial  Correlation  and  the  Item  Reliability 
Index  show  that  the  item  discriminated  quite  acceptably  between  'upper' 
and  'lower'  students. 

The  detailed  statistics  indicate  that  each  of  the  distractors  was  a 
plausible  answer  attracting  more  of  the  poorer  students  than  the  better 
students. 


This  item  thus  proved  effective  in  its  present  form. 
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EVALUATION  3  —  TEST  ITEM  36 


36. 


We  all  are  blind  until  we  see 

That  in  the  human  plan 
Nothing  is  worth  the  making  if 

It  does  not  make  the  man. 

Why  build  these  cities  glorious 

If  man  unbuild ed  goes? 
In  vain  we  build  the  world,  unless 

The  builder  also  grows. 

What  is  the  poet»s  conclusion? 

A.  We  are  all  backward  until  we  know  both  city  and 
country  life. 

B.  Character  building  should  not  be  neglected. 

C.  Man  grows  when  cities  are  built  and  grow  large. 

D.  In  the  human  plan,  the  urban  population  are  supposed 
to  grow. 


JITEM 

N 

OMIT              DIF 

NR 

NF 

K 

1 

© 

3 

4 

;      36 

1026 

7 

0.664 

0 

2 

63 

6§1 

205 

70 

188 

UPPER 

5 

0 

4 

176 

7 

1      i 

210 

UPPER 

4 

1 

5 

181 

18 

5      i 

2U 

UPPER 

3 

1 

11 

152 

38 

12 

196 

UPPER 

2 

2 

11 

101 

63 

19 

218 

UPPER 

1 

3 

32 

71 

79 

33 

TEST 

SCORE 

MEANS 

39.7 

51.8 

41.8 

39.8 

Z-SCORE  MEANS 

0.42 

-0.75 

0.33 

-0.54 

-0.75 

BISERIAL  C0RREL  #  0.605 
ITEM  REL.  INDEX  #  0.286 


This  item  was  designed  to  test  the  student's  ability  to  interpret  or  to 
grasp  the  thought  of  a  work  as  a  whole.   It  has  been  placed  in  the 
Comprehension  category. 

The  item  was  easy  with  approximately  66$  of  the  students  responding 
correctly.  The  Biserial  Correlation  indicates  that  the  item  discriminated 
exceptionally  well.  At  the  same  time,  the  Item  Reliability  Index  shows 
that  one  can  depend  upon  the  Biserial  Correlation  with  a  high  degree  of 
confidence. 


The  remaining  data  indicate  that  very  few  upper  students  chose  distractors, 
yet  each  attracted  the  lower  students.  Thus  all  distractors  functioned 
repsonebly  well,  with  alternative  3  being  the  strongest.  This  is  the 
most  literal  of  all  the  distractors  and  obviously  was  extremely  attractive 
to  the  week  students.  Apparently  lower  students,  tending  to  accept  the 
most  literal  meaning,  found  it  difficult  to  interpret  the  selection  and 
grasp  the  central  thought. 

The  entire  item  functioned  exceptionally  well,  and  might  be  used  as  a 
prototype  to  teach  weaker  students  how  to  interpret  varying  literary 
selections. 
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EVALUATION  4  —  TEST  ITEM  93 

Read  the  following  quotation  and  answer  question  93* 

"Some  books  are  to  be  tasted,  others  to  be  swallowed,  and  some  few  to  be 
chewed  and  digested." 

93.   "to  be  tasted"  means  to  be 

A.  studied  carefully 

B.  sampled 

C.  disregarded 
D •  skimmed 


ITEM           N       OMIT 

DIF 

NR 

NF 

K 

93       1026         1 

0.496 

23 

4 

188            UPPER 

5 

0 

210            UPPER 

4 

0 

214            UPPER 

3 

8 

196            UPPER 

2 

4 

218            UPPER 

1 

12 

TEST  SCORE  MEANS 

Z-SCORE  MEANS            -0.01 

BISERIAL  CORREL  #|0.195| 

1 

2 

3 

(4) 

69 

426 

\W\ 

497 

2 

176 

0 

110 

5 

m 

3 

103 

11 

82 

1 

112 

17 

1           ° 

93 

34 

87 

6 

79 

38.8   48.5   37.0   49.9 
-0.85   0.07  -1.01   0.15 


This  item  was  placed  in  the  comprehension  category  because  the  student  must 
translate  an  abstraction  into  literal  terms. 

As  approximately  50$  of  the  students  selected  the  correct  answer,  the  item 
was  of  acceptable  difficulty.   The  Biserial  Correlation,  however,  is  not 
acceptable,  indicating  that  the  item  did  not  discriminate  well  between  upper 
and  lower  students. 

The  Keyed  Answer  (alternative  4)  and  distractor  1  show  good  discrimination 
between  upper  and  lower  students.   However,  distractor  2  operated  poorly, 
attracting  more  of  the  upper  students  than  lower  students.   Perhaps  students 
selected  "sampled"  as  a  better  literal  translation  than  "skimmed".   As  this 
choice  can  be  defended,  possibly  the  item  has  no  best  answer.   If  a  new 
word,  less  closely  related  in  meaning  to  "skimmed"  were  substituted  for 
"sampled",  this  item  would  be  strengthened.  Distractor  3>  almost  totally 
ineffective,  must  be  made  more  plausible  to  lower  students.""' 


The  suggested  revisions  (if  successful)  would  produce  the  following 
•chain  reaction'  effect  on  the  statistics  when  the  new  version  is 
tested. 

1)  If  distractor  2  were  changed  to  make  it  more  obviously  wrong  to 
upper  students,  yet  still  plausible  to  lower  students,  more  upper  stu- 
dents than  lower  students  would  be  directed  away  from  it.   The  discrim- 
inatory power  of  distractor  2  would  then  be  increased.  Further,  most 
of  the  upper  students  formerly  choosing  2  would  change  to  the  keyed 
answer,  thus  making  alternative  4  more  discriminating. 

2)  If  distractor  3  were  changed  to  make  it  appear  more  plausible  to 
only  the  lower  students,  then  the  discriminatory  power  of  this  distrac- 
tor would  be  increased.   Many  of  these  lower  students  would  be  drawn 
from  those  formerly  choosing  the  keyed  answer,  thus  making  alternative 
4  even  more  discriminating. 

3)  Improving  the  discriminatory  power  of  these  three  alternatives 
would  improve  the  discriminatory  power  of  the  entire  item  and  substan- 
tially increase  the  Biserial  Correlation  and  Item  Reliability  Index. 
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EVALUATION  5  —  TEST  ITEM  17 

17.  nAll  honor  to  him  who  shall  win  the  prize, n 
The  world  has  cried  for  a  thousand  years; 
But  to  him  who  tries  and  fails  and  dies 
I  give  great  honor  and  glory  and  tears. 

Which  of  the  following  selections  from  the  text  expresses 
similar  thought? 

A.  The  Memorial  Cup  Series 

B.  The  Service 

C.  At  the  Cedars 

D.  Skeleton  in  Armour 


ITEM 

N 

OMIT      DIF 

NR 

NF   K 

1 

® 

3 

"  4  1 

17 

1026 

204     0.534 

0   2 

81 

548 

\M 

H9 

188 

UPPER 

5 

30 

2 

141 

7 

8 

210 

UPPER 

4 

34 

9 

143 

3 

21 

2U 

UPPER 

3 

61 

16 

103 

9 

25 

196 

UPPER 

2 

37 

23 

90 

11 

35 

218 

UPPER 

1 

42 

31 

71 

14 

60 

TEST 

SCORE 

MEANS 

40.8 

51.3 

43.9 

42.4 

Z-SCORE  MEANS      0.09 

-0.66 

0.24 

-0.36 

-0.54 

BISERIAL  CORREL  #  0.385 

i 

ITEM 

REL. 

INDEX  #  0.192 

i 

To  answer  the  item  correctly,  the  student  must  first  comprehend  the  given 
passage,  analyze  it  for  theme,  and  then  relate  its  theme  to  a  textbook 
selection.  This  item  was  placed  in  the  Higher  Mental  Processes  category. 

The  level  of  difficulty  indicates  that  slightly  more  than  half  the  stu- 
dents selected  the  correct  answer.   In  this  respect  the  item  was  acceptable, 
The  Biserial  Correlation  indicates  that  the  item  discriminated  quite  well 
since  those  who  responded  correctly  scored  higher  on  the  total  test.  Those 
who  responded  incorrectly  scored  lower  on  the  total  test.  The  Item  Relia- 
bility Index  indicates  that  one  can  depend  on  this  Biserial  Correlation 
with  a  fairly  high  degree  of  confidence. 

The  distractors  functioned  quite  effectively  with  distractor  4  being  the 
most  successful.   Students  who  chose  this  alternative  failed  to  analyze 
the  passage  for  theme  and  probably  responded  to  surface  words  such  as  "for 
e  thousand  ye?rsn.  The  weakest  of  the  distractors  is  alternative  3.  This 
--lternative  was  probably  rejected  because  At  the  Cedars  contains  no  men- 
tion of  a  contest  and  this  idea  runs  throughout  the  given  selection. 

Basically  the  item  functioned  well,  but  it  could  be  improved.  Alternative 
3  could  name  a  selection  containing  the  idea  of  a  contest  (e.g.,  The 
Divinp  Fool) . 

*  To  facilitate  identification,  unacceptable  elements  are  boxed. 
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EVALUATION  6  —  TEST  ITEM  63 


Reed  the  three  stanzas  below  and  answer  question  63 

A.  To  startled  skies  the  pibroch  sounds, 
The  frenzied  chargers  champ  and  neigh; 
With  fearful  impact  battle  joins: 

And  stark  death  rides  the  wind  today. 

B.  The  blaring  tocsin  calls  to  battle, 
Claymore  thumps  on  claymore; 

Shot  and  shell  on  helmets  rattle 
Like  pebbles  on  the  sea-shore:  ... 

C.  The  shattering  trumpet  shrilleth  high, 
The  nard  brands  shiver  on  the  steel, 

The  splintered  spear-shafts  crack  and  fly, 
And  horse  and  rider  reel:  ... 


63.  In  which  stanza  are  the  images  most 

confused? 

ITEM     N   OMIT      DIF      NR   NF 
63   1026    8      0.368          0 
188     UPPER      5    0 
210     UPPER      4    1 
214     UPPER      3    1 
196     UPPER      2    2 
218     UPPER      1    4 

K      © 

1    W) 

84 
86 
72 
21. 

2     3 

354    271 
59    06 

66  £8 

67  55 
73     43 
89    49 

15 

0 

1 

5 

6 

3 

43.2 

-0.45 

TEST  SCORE  MEANS 
Z-SCQRE  MEANS 

-0.34 

48.5 
-0.0] 

»   47.1  \l 
.    -0.10   c 

•-9.7I 

).15 

BISERIAL  CCRREL  # 
ITEM  REL.  INDEX  # 

0.030 

0.015 

This  item  has  been  classified  as  testing  the  Higher  Mental  Processes.  The 
student  is  asked  to  analyze  and  evaluate  three  stanzas  and  decide  in  which 
stanza  the  images  are  most  confused. 

The  level  of  difficulty  indicates  that  the  students  found  this  item  diffi- 
cult.  The  Biserial  Correlation  is  not  acceptable.  The  item  does  not 
discriminate  well  between  upper  and  lower  students;  that  is,  lower  students 
tended  to  answer  this  item  correctly  as  often  as  the  upper  students.   Since 
the  Biserial  Correlation  is  unacceptable  the  Item  Reliability  Index  is  to 
be  disregarded. 

There  ere  several  weaknesses  in  this  item.   In  the  stem,  the  word  "confused" 
was  probably  misinterpreted  to  mean  something  other  than  disparate  or  unre- 
lated which  the  examiners  evidently  intended  it  to  mean.  The  attractiveness 
of  the  distractors,  especially  alternative  3,  would  indicate  that  many  of 
the  upper  students  accepted  the  word  "confused"  as  referring  to  a  physical 
scene  rather  than  to  a  number  of  unrelated  images.   If  the  word  "unrelated" 
were  substituted  for  "confused",  the  item  should  prove  more  useful. 

The  vocabulary  in  the  stanzas  which  served  as  the  alternatives  was  probably 
too  difficult  for  Grade  H  students.   Such  terms  as  "pibroch"  and  "claymore" 
are  beyond  the  vocabulary  of  the  average  student  in  this  group.  Material 
containing  vocabulary  of  such  complexity  must  be  used  with  caution. 

The  use  of  only  three  alternatives  caused  difficulty  for  some  students. 
This  may  have  been  avoided  had  the  word  "three"  in  the  directions  been 
capitalized,  or  if  the  stem  had  read  "In  which  of  the  THREE  stanzas...." 


EVALUATION  7  —  TEST  ITEM  27 

(Evaluations  7,  8,  and  9  deal  with  a  block  of  items 
ell  based  on  the  poem  given  on  this  page). 

The  following  poem  applies  to  questions  27  -  30. 

A  noiseless  patient  spider, 
I  marked  where  on  a  little 

promontory  it  stood  isolated, 
Marked  how  to  explore  the  vacant 

vast  surrounding, 
It  launched  forth  filament,  filament, 

filament,  out  of  itself. 
Ever  unreeling  them,  ever  tirelessly 

speeding  them. 
And  you  0  my  soul  where  you  stand, 
Surrounded,  detached,  in  measureless 

oceans  of  space. 
Ceaselessly  musing,  venturing,  throw- 
ing, seeking  the  spheres  to 

connect  them. 
Till  the  bridge  you  will  need  be  form'd 

till  the  ductile  anchor  hold, 
Till  the  gossemer  thread  you  fling 

catch  somewhere, 

0  my  soul. 

27.  The  above  poem  is  BEST  classified  as 

A .  narrative 

B.  dramatic 

C.  free  verse 

D.  ode 
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—  27,  28,  and  29 


ITEM           N       OMIT              DIF 

NR 

NF       K 

1 

2 

© 

4 

27       1026         5             0.619 

0       3 

115 

171 

535 

LOO 

188              UPPER 

5 

0 

9 

?/31 

•\34 

?{124 

*  1124 

24 

210             UPPER 

4 

0 

23 

29 

214             UPPER 

3 

0 

U 

38 

147 

15 

196             UPPER 

2 

1 

32 

?/33 
135 

47.7 

9  (116 
"   1124 
48.8     | 

14 

218             UPPER 
TEST  SCORE  MEANS 

1 

4 

37 

43.7 

18 

50. 

51 

Z -SCORE  MEANS              0.30 

-0.36 

-0.01 

0.07       ( 

).24 

BISERIAL  C0RREL  #|0.101| 

ITEM  REL.    INDEX  #  0.049 

This  item,  designed  to  test  familiarity  with  the  forms  of  poetry,  required 
the  student  to  recall  facts  about  the  form  of  free  verse.  62$  of  the  stu- 
dents answered  this  Knowledge  item  correctly.  Although  this  level  of 
difficulty  is  acceptable,  the  Biserial  Correlation  indicates  that  serious 
flaws  exist  in  the  item. 
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(Evaluation  7  -  cont'd) 

The  correct  alternative  was  ineffective  since  virtually  as  many  lower 
students  as  upper  students  answered  correctly.  The  test  score  mean 
(average)  of  those  selecting  this  correct  alternative  is  lower  than  the 
test  score  mean  of  those  choosing  distractor  4.  Too  many  upper  students 
were  also  attracted  to  distractor  2. 

This  situation  could  have  been  caused  by  the  format*  of  the  poem.  Further 
the  word  "0"  could  have  been  accepted  as  a  false  clue  to  distractor s  2  and 
4.  It  may  be,  however,  that  more  explicit  definitions  of  the  terms  used 
in  describing  various  forms  of  poetry  should  become  the  teacher's  concern. 

EVALUATION  8  —  TEST  ITEM  28 

28.  What  is  compared  to  the  patient  spider? 

A.  a  bridge  and  anchor 

B.  the  human  soul 

C.  gossamer  thread 

D.  measureless  oceans  of  space 


ITEM      N 

OMIT      DIF 

NR 

NF 

K 

1 

<k) 

3 

4   ' 

28    1026 

5     0.739 

0 

2 

108 

758 

1  35  | 

120 

188 

UPPER 

5 

0 

10 

173 

0 

5  1 

210 

UPPER 

4 

1 

10 

187 

2 

10 

214 

UPPER 

3 

0 

22 

168 

5 

19    ; 

196 

UPPER 

2 

1 

36 

120 

9 

30 

218 

UPPER 

1 

3 

30 

110 

19 

56 

TEST  SCORE 

MEANS 

43.2 

50.6 

37.8 

40.8  ! 

Z-SCORE  MEANS      0.64 

-0.45 

0.24 

-0.93 

-0.65 

BISERIAL  CORREL  #  0.508 

ITEM  REL. 

INDEX  #  0.223 

To  answer  the  item  correctly,  the  student  is  required  to  comprehend  the 
significance  of  the  words  of  a  poem  with  reference  to  their  context  in  order 
to  grasp  the  central  thought  of  the  poem.  He  must  perceive  a  relationship 
between  a  spider  patiently  spinning  his  web  and  a  figurative  description  of 
the  soul  groping  in  the  world  of  human  experience.  The  item  was  placed  in 
the  Higher  Mental  Processes  category. 

As  74$  of  the  students  selected  the  correct  answer,  the  item  was  very  easy. 
The  Biserial  Correlation  indicates  that  the  item  discriminated  exceptionally 
well  between  upper  and  lower  students. 

Each  alternative  also  discriminated  well.  However,  distractor  3  was  not 
entirely  acceptable,  attracting  less  than  5%   of  the  students.  To  improve 
the  item,  this  alternative  should  be  made  more  attractive  to  more  lower 
students.  This  revision  would  make  the  item  somewhat  more  difficult,  and 
thus  decrease  the  difficulty  index. 


The  format  of  the  poem  used  for  this  block  of  items  results  in  a  cramped 
appearance  and  an  unnatural  breaking  of  the  poetic  lines.  This  is  due 
to  the  double  columned  format  of  the  examination  booklet. 
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EVALUATION  9  —  TEST  ITEM  29 


29.  The  poem  illustrates  a  single  sustained 

A.  simile 

B.  hyperbole 

C.  personification 

D.  metaphor 


[item    n  omit    dif 

NR 

NF 

K 

1 

2 

3 

© 

29   1026   15     0.389 

0 

4 

102 

138 

372 

399 

188     UPPER 

5 

1 

21 

13 

45 

108 

210      UPPER 

4 

4 

20 

19 

75 

92 

214      UPPER 

3 

3 

IC 

29 

85 

77 

196      UPPER 

2 

3 

19 

31 

80 

63 

218      UPPER 

1 

4 

22 

46 

87 

59 

TEST  SCORE  MEANS 

48.0   44.3 

46.7 

51.0 

Z-SCORE  MEANS    -0.28 

-0.01  -0.36 

-0.10 

0.24 

BISERIAL  CORREL  #|0.267| 

ITEM  REL.  INDEX  #  0.130 
1 

The  item  requires  a  knowledge  of  terminology  and  the  application  of  this 
knowledge  to  a  passage.  Analysis  of  the  poem  is  also  required  to  per- 
ceive a  relationship  between  a  spider  and  the  human  soul  as  suggested  by 
the  poet.  The  item  was  placed  in  the  Higher  Mental  Processes  category. 

The  difficulty  level  indicates  that  the  item  proved  to  be  difficult. 

The  Biserial  Correlation  indicates  that  the  item  did  not  discriminate  well 

between  upper  and  lower  students.  Therefore  the  item  is  unacceptable. 

The  weakness  of  the  item  becomes  apparent  when  the  alternatives  are 
examined.  Distractor  1  is  of  little  value  as  it  attracted  an  equal 
number  of  upper  and  lower  students.  Distractor  3  is  too  attractive  to 
upper  students.  The  low  Biserial  Correlation  reflects  the  low 
discriminatory  powers  of  these  alternatives. 

In  its  present  form  this  item  is  not  effective.  To  be  more  acceptable 
distractors  might  be  revised  by  substituting  all  alternatives  with  des- 
criptive phrases. 
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EVALUATION  10  —  TEST  ITEM  97 

97.  The  air  is  still  and  the  lake  calm. 

Which  of  the  following  sentences  says  this  most  effectively,  the 
words  reinforcing  the  sense? 

A.  I  hear  lake  waters  lisping  as  they  lazily  strike  the  shore. 

B.  I  hear  lake  water  lapping  with  low  sounds  by  the  shore. 

C.  I  hear  the  waters  of  the  lake  washing  on  the  shore  faintly. 

D.  I  hear  the  lake's  waters  splashing  on  the  still  shore. 


ITEM    N   OMIT 

DIF 

NR 

NF   K 

(2 

2 

3 

4 

97  1026   39 

0.519 

0   1 

533 

172 

207 

75 

188     UPPER 

5 

2 

141 

16 

26 

3 

210    UPPER 

4 

2 

132 

30 

37 

9 

214     UPPER 

3 

12 

123 

25 

45 

9 

196    UPPER 

2 

5 

88 

44 

40 

19 

218     UPPER 

1 

18 

49 

57 

59 

35 

TEST  SCORE  MEANS 

51.9 

44.4 

46.0 

40.3 

Z-SCORE  MEANS 

0.05 

0.33 

-0.36 

-0.20 

-0.75 

BISERIAL  CORREL  #  0.445 

ITEM  REL.  INDEX  #  0.222 

It  nay  be  maintained  that  this  item  tests  simply  an  awareness  of 
aesthetic  elements  in  literature.  However,  the  item  does  require  the 
student  to  evaluate  in  terms  of  internal  evidence  since  he  must  decide 
in  which  sentence  the  words  most  effectively  reinforce  the  desired  sense , 
Thus  the  item  has  been  placed  in  the  Higher  Mental  Processes  category. 

The  item  is  of  average  difficulty  and  discriminates  well.  All  the 
alternatives  work  very  effectively.  These  data  indicate  that  this  is 
an  excellent  item. 
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COMMENTS 

1.  In  item  63  several  students  marked  a  fourth  alternative  when 
the  item  contained  only  three  possible  alternatives.  Teachers 
should  caution  students  against  assuming  that  there  is  a  set 
number  of  alternatives  in  multiple-choice  test  items.  While 
four  alternatives  are  most  common  with  multiple-choice  items, 
three  or  five  are  also  used. 

2.  In  item  93,  it  can  be  argued  that  there  is  no  one  correct 
answer.  This  was  evident  from  the  analysis  when  many  upper 
students  found  an  incorrect  alternative  very  attractive. 
There  must  always  be  a  definitely  best  answer  to  each  item  and 
the  possibility  of  dispute  should  be  held  to  a  minimum.  The 
teacher  must  also  be  certain  that  vocabulary  is  clear  and  con- 
cise and  not  misleading  to  the  student. 

3.  It  is  well  to  keep  in  mind  that  unacceptable  statistics  need 
not  always  be  the  result  of  a  poorly  constructed  item.  When  the 
analysis  indicates  that  an  item  has  not  operated  well,  the 
teacher  must  consider  all  the  following  areas  when  determining 
the  source  of  the  problem: 

a)  technical  quality  if  the  item  (of  the  stem  and  any 
materials  upon  which  the  item  is  based,  as  well  as 
of  the  individual  alternatives) 

b)  appropriateness  and  significance  of  the  concept 
that  he  is  testing 

c)  instructional  procedures  that  he  has  used  in  teaching 
the  concept 

4.  Many  teachers  have  had  little  training  in  the  field  of 
statistics.  For  this  reason,  the  present  booklet  purposely 
avoids  technical  explanations  that  may  be  confusing.  One 
evident  example  of  this  is  found  in  the  explanation  of  the 
Item  Reliability  Index.  Completely  disregarding  the  source 
from  which  this  index  is  derived, only  the  most  practical 
consequence  of  its  use  in  interpreting  an  item  analysis  is 
stated  in  this  booklet  (see  CHART  3). 
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5.  In  testing  Literature  IX,  the  Knowledge  level  (memorization) 
is  often  stressed.  Present  practices,  however,  indicate  a 
shift  of  emphasis  from  simple  recall  and  a  preoccupation  with 
specific  works  to  the  student's  ability  to  understand  and  use 
literary  concepts.  Releases  from  the  Department  of  Education 
indicate  that  the  1968  Grade  IX  Departmental  Examination  will 
not  include  items  from  the  text  and  that  80$  of  the  test  will 
focus  on  items  testing  Higher  Mental  Processes  beyond  the 
Knowledge  level.  This,  however,  should  not  be  misconstrued  to 
mean  that  the  Knowledge  level  is  to  be  neglected.  Rather,  it 
is  to  be  considered  as  a  basis  for  the  development  of  abilities 
in  understanding  and  using  literary  concepts  at  a  higher  mental 
level. 

6.  Computer  services  may  not  be  readily  available  to  many  teachers. 
Where  this  is  the  case  these  teachers  may  wish  to  obtain  the 
pamphlet,  Short-cut  Statistics  for  Teacher-made  Tests,  pub- 
lished by  the  Educational  Testing  Service,  Berkeley,  California. 
The  teacher  is  shown  how  he  may,  within  his  own  classroom  situa- 
tion, roughly  approximate  some  of  the  information  obtainable 
from  the  computer  analysis  dealt  with  in  the  present  booklet. 

7.  Periodically  the  computer  program  for  item  analysis  is 
revised  to  include  additional  information  or  slightly  change 
the  format.  For  this  reason  future  computer  'print-outs' 
may  vary  slightly  from  the  ones  shown  in  this  booklet.  Re- 
cent changes  have  included  the  addition  of  the  Item  Relia- 
bility Index  (incorporated  in  the  analyses  appearing  through- 
out this  booklet),  and  the  total  number  of  students  in  each 
of  the  five  groups  as  indicated  in  the  column  under  'N'  of 
Charts  2,  5,  and  6.  A  current  change  (not  shown  in  this 
booklet)  is  the  use  of  the  word  'GROUP'  to  replace  the  word 
'UPPER'  as  it  appears  in  all  the  foregoing  item  analyses. 
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